Skip to content

Add Weibo API scheme for JSON profile extraction#214

Merged
soxoj merged 5 commits intomasterfrom
copilot/add-weibo-support
Apr 4, 2026
Merged

Add Weibo API scheme for JSON profile extraction#214
soxoj merged 5 commits intomasterfrom
copilot/add-weibo-support

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Mar 29, 2026

Existing Weibo scheme relied on HTML parsing with $CONFIG flags, requiring cookies and failing on redirects. Weibo exposes a JSON API at /ajax/profile/info that returns structured user data without authentication.

Changes

  • New "Weibo API" scheme with extract_json: True and two url_mutations:
    • /clairekuo/ajax/profile/info?custom=clairekuo
    • /u/6215884155/ajax/profile/info?uid=6215884155
  • Extracts 12 fields: weibo_id, username, fullname, bio, image, gender, location, verified, verified_reason, follower_count, following_count, statuses_count
  • Existing HTML-based scheme preserved as fallback (distinct flags, no conflict)
  • E2e tests added in tests/test_e2e.py: test_weibo_api (by username) and test_weibo_api_by_id (by user ID), marked @pytest.mark.github_failed since Weibo may block CI servers

Example API response shape

{"ok":1,"data":{"user":{"idstr":"1733299783","screen_name":"郭靜Claire","domain":"clairekuo","gender":"f","location":"台湾 台北市","followers_count":3126727,...}}}

📱 Kick off Copilot coding agent tasks wherever you are with GitHub Mobile, available on iOS and Android.

Copilot AI linked an issue Mar 29, 2026 that may be closed by this pull request
Copilot AI changed the title [WIP] Add Weibo support with user details Add Weibo API scheme for JSON profile extraction Mar 29, 2026
Copilot AI requested a review from soxoj March 29, 2026 17:14
@soxoj soxoj marked this pull request as ready for review March 29, 2026 20:41
@soxoj
Copy link
Copy Markdown
Owner

soxoj commented Mar 29, 2026

@copilot make e2e-test

Copy link
Copy Markdown
Contributor Author

Copilot AI commented Mar 29, 2026

@copilot make e2e-test

Added two e2e tests in 6d83d75:

  • test_weibo_api — fetches profile by username (/clairekuo) via mutate_url/ajax/profile/info?custom=clairekuo
  • test_weibo_api_by_id — fetches profile by user ID (/u/6215884155) via mutate_url/ajax/profile/info?uid=6215884155

Both are marked @pytest.mark.github_failed since Weibo may block CI servers.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • weibo.com
    • Triggering command: /usr/bin/python python -m pytest tests/test_e2e.py::test_weibo_api tests/test_e2e.py::test_weibo_api_by_id -v (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

@soxoj
Copy link
Copy Markdown
Owner

soxoj commented Apr 4, 2026

@copilot resolve the conflicts

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 4, 2026

@copilot resolve the conflicts

Resolved the merge conflict in tests/test_socid_improvements.py in 1e695ce — kept both the Weibo API tests and all new tests from master. All 81 unit tests pass.

Copilot finished work on behalf of soxoj April 4, 2026 17:13
@soxoj
Copy link
Copy Markdown
Owner

soxoj commented Apr 4, 2026

@copilot mark e2e test_d3 as github_failed

Agent-Logs-Url: https://github.com/soxoj/socid-extractor/sessions/d7180735-49cc-4aa4-ab8e-85ab95b36a08

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 4, 2026

@copilot mark e2e test_d3 as github_failed

Done in c960c3f — added @pytest.mark.github_failed to test_d3.

Copilot finished work on behalf of soxoj April 4, 2026 18:30
@soxoj soxoj merged commit ba162bc into master Apr 4, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Weibo

2 participants